{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise - Feature Importance and Feature Selection\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, you will train a base model and investigate which features the performance of the model seems to be driven by. Then you will apply feature selection techniques to reduce the feature set and investigate the effect this has on the model's performance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup, Baseline Model and Baseline Performance Score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the cells below to create a synthetic dataset for binary classification with 50 features and 10,000 examples. Imagine the target `y` is the direction of price movements which we would like to predict using the 50 features at our disposal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - create dataset and display basic statistics\n",
    "from sklearn.datasets import make_classification\n",
    "X, y = make_classification(n_samples=10_000, n_classes=2, n_features=50, n_informative=10, n_redundant=10, class_sep=0.4, n_clusters_per_class=3, random_state=52)\n",
    "\n",
    "X = pd.DataFrame(X)\n",
    "y = pd.Series(y)\n",
    "\n",
    "# DO NOT MODIFY - Split the data into training and test sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we continue, we would like to establish a baseline score. We will choose accuracy as the relevant performance metric.  \n",
    "Write code to calculate and display the accuracy score on the _test set_ of a naive baseline model that always predicts the majority class (based on the majority class in the _training set_).\n",
    "\n",
    "> **HINT:**\n",
    "> First, you have to find the majority class (either `0` or `1`) in the target variable on the _training set_. You can use `df.value_counts()` or you can look at the `mode()` of the target, since it only has two classes.  \n",
    "> Next, create an array with the same length as `y_test` with all elements equal to the majority class you just found.  \n",
    "> Finally, use this as the vector of predictions to evaluate this naive baseline model on the _test set_.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# FILL IN - Find the majority class in the training set\n",
    "\n",
    "# FILL IN - Calculate the precision of the majority class classifier on the test set\n",
    "baseline_test_acc = ...\n",
    "baseline_test_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Basic Feature Selection with Permutation Importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below to train a `LogisticRegression` model with its default hyperparameter values, using all 50 features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - import and train a LogisticRegression\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "clf = LogisticRegression(random_state=52)\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cells below to get the cross-validated accuracy score and the actual accuracy score on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY\n",
    "from sklearn.model_selection import cross_val_score\n",
    "cross_val_score(clf, X_train, y_train, cv=5, scoring=\"accuracy\").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY -\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "y_pred = clf.predict(X_test)\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the permutation importance scores of the features using the test set. Use `n_repeats=10` and `random_state=52`. Store the average permutation importance scores in `mean_perm_imps`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - import\n",
    "from sklearn.inspection import permutation_importance\n",
    "\n",
    "# FILL IN - Calculate permutation importance scores for the features in the test set\n",
    "# Use n_repeats=10 and random_state=52\n",
    "perm_imps = ...\n",
    "mean_perm_imps = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to print out the features listed in decreasing order of absolute value of mean permutation importance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - Sort and print the features by decreasing absolute permutation importance\n",
    "sorted_idx = np.argsort(np.abs(mean_perm_imps))[::-1]\n",
    "for i in sorted_idx:\n",
    "    print(f\"{i}: {mean_perm_imps[i]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reduce the feature set by dropping features that have a permutation importance score less than `0.003`. Store the resulting reduced feature sets in `X_train_reduced` and `X_test_reduced`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Filter out features with permutation importance less than 0.003\n",
    "\n",
    "X_train_reduced = ...\n",
    "X_test_reduced = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to see how many features remain. (There should be 9.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - There should be 9 features remaining\n",
    "X_train_reduced.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-train the classifier from earlier and check its average cross-validated accuracy and test accuracy scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Train a new LogisticRegression model on the reduced feature set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Calculate the mean cross-validated accuracy of the new model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Calculate the precision of the new model on the test set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** Reducing the feature set may or may not improve performance. After all, even some less \"important\" features still provide some information and eliminating them might result in a hit to performance scores. But a reduction in performance may still be worthwhile if it means faster model training and a more interpretable model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
